Display of a live scene and auxiliary object

ABSTRACT

A mobile device comprises one or more processors, a display, and a camera configured to capture an image of a live scene. The one or more processors are configured to determine a location of the mobile device and display an augmented image based on the captured image. The augmented image includes at least a portion of the image of the live scene and a map including an indication of the determined location of the mobile device. The one or more processors are also configured to display the at least a portion of the image of the live scene in a first portion of the display and displaying the map in a second portion of the display. The augmented image is updated as the mobile device is moved, and the map is docked to the second portion of the display as the augmented image is updated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. Non-Provisional Application No.17/357,447, filed on Jun. 24, 2021, and titled “DISPLAY OF A LIVE SCENEAND AUXILIARY OBJECT,” which is a continuation of U.S. Non-ProvisionalApplication No. 16/668,878, filed on Oct. 30, 2019, and titled “DISPLAYOF A LIVE SCENE AND AUXILIARY OBJECT,” which is a continuation of U.S.Non-Provisional Application No. 15/721,095, filed on Sep. 29, 2017, andtitled “DISPLAY OF A LIVE SCENE AND AUXILIARY OBJECT,” which are herebyincorporated by reference in their entirety for all purposes.

BACKGROUND 1. Field

The subject matter disclosed herein relates to displaying data forusers, including displaying augmented reality data to augment a livescene, maps, personal and other communications, and other data.

2. Background

Users can capture images or video of a live scene using a camera of amobile device and can view the captured images or video of the livescene on a display of the mobile device. However, there are noapplications that generate an augmented image for presentation to theuser that may include additional information not already within thecaptured live scene. Hence, there is interest for applications thatprovide the user with useful information by displaying an augmentedimage.

SUMMARY

An example method for displaying, in a mobile device, an augmented imagecan include capturing, by a camera of the mobile device, an image of alive scene; displaying the augmented image based on the captured image,the augmented image including at least one auxiliary object docked to afeature comprising a physical object visible in the captured image; andupdating the augmented image as the mobile device is moved, wherein theat least one auxiliary object remains docked to the feature as theaugmented image is updated. The method can optionally further includereceiving an input designating the feature for docking the at least oneauxiliary object; and displaying the auxiliary object docked to thedesignated feature. The method can also optionally further includedetermining a location of the mobile device. The method can include,responsive to the feature no longer being visible in the image due tomovement of the mobile device, either removing the at least oneauxiliary object from the augmented image or displaying the at least oneauxiliary object to another feature that is visible in the capturedimage. The auxiliary object can include a top-down map with anindication of the determined location of the mobile device. Theauxiliary object docked to the feature can be independently scalablewhile docked to the feature. The method can further include displaying,in addition to the at least one auxiliary object, system generatedaugmented reality data, which can include, for example, tags determinedbased on the determined location of the mobile device.

An example mobile device can include a camera configured to capture animage of a live scene, a display, a memory, and one or more processorscoupled to the camera, the display, and the memory of the mobile device.The one or more processors can be configured to present an augmentedimage on the display based on the captured image, the augmented imageincluding at least one auxiliary object docked to a feature comprising aphysical object visible in the captured image. The one or moreprocessors can further be configured to update the augmented image asthe mobile device is moved, wherein the auxiliary object remains dockedto the feature as the augmented image is updated. Furthermore, thedisplay of the mobile device can comprise a touchscreen displayconfigured to provide an input designating the feature for docking theat least one auxiliary object. Responsive to the input, the one or moreprocessors can be configured to display the auxiliary object docked tothe designated feature. To provide the input designating the feature,the touchscreen display can be configured to receive a tap gesture or adrag gesture by a user. Furthermore, the mobile device can include acommunications subsystem configured to communicate with one or moreservers to receive data for displaying the auxiliary object or fordisplaying augmented reality data. Additionally, the mobile device caninclude a location engine configured to determine a location of themobile device, where the determined location is displayed on a mapdisplayed as the auxiliary object. The one or more processors of themobile device can further be configured to, responsive to the feature nolonger being visible in the image due to movement of the mobile device,remove the at least one auxiliary object from the augmented image, ordock the at least one auxiliary object to another feature that isvisible in the captured image.

An example apparatus for displaying an augmented image can include meansfor capturing an image of a live scene; means for displaying theaugmented image based on the captured image and means for updating theaugmented image. The means for displaying the augmented image caninclude means for displaying the augmented image such that the augmentedimage includes at least one auxiliary object docked to a featurecomprising a physical object visible in the captured image. The meansfor updating the augmented image can include means for updating theaugmented image such that, as the mobile device is moved, the at leastone auxiliary object remains docked to the feature as the augmentedimage is updated. The apparatus can further include means for receivingan input designating the feature for docking the at least one auxiliaryobject. The means for receiving the input designating the feature cancomprise means for receiving a tap gesture or a drag gesture todesignate the feature. The apparatus can further include means fordetermining a location of the mobile device, and the auxiliary objectcan be associated with the determined location. In one example, themeans for displaying the augmented image comprises means for displayinga top-down map associated with the determined location. The means forupdating the augmented image can comprise means for removing the atleast one auxiliary object responsive to the feature no longer beingvisible in the captured image due to movement of the mobile device.Alternatively, the means for updating the augmented image can comprisemeans for docking the at least one auxiliary object to another featurethat is visible in the captured image responsive to the feature nolonger being visible in the captured image due to movement of the mobiledevice.

An example of a non-transitory computer readable medium storinginstructions for execution by one or more processors of a mobile deviceto perform operations comprising capturing, using a camera of the mobiledevice, an image of a live scene; displaying, on a display of the mobiledevice, an augmented image based on the captured image, the augmentedimage including at least one auxiliary object docked to a featurecomprising a physical object visible in the captured image; and updatingthe augmented image on the display of the mobile device as the mobiledevice is moved, wherein the auxiliary object remains docked to thefeature as the augmented image is updated. In one implementation, thenon-transitory computer readable medium further includes instructions toprocess an input designating the feature for docking the at least oneauxiliary object and instructions to display the auxiliary object dockedto the designated feature.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a user navigating a venue.

FIG. 1B illustrates a top-down map showing a location of the user insidethe venue.

FIG. 1C illustrates an image of a mobile device camera view of a livescene displayed on the screen of a mobile device where the image of themobile device camera view is augmented with augmented reality data.

FIG. 2A illustrates an image of a mobile device camera view of a livescene displayed on the screen of a mobile device where the image isaugmented with augmented reality data while an auxiliary object isdocked to a feature visible in the image.

FIG. 2B illustrates an image of a mobile device camera view of a livescene similar to that illustrated in FIG. 2A where the auxiliary objectis docked to a different feature in the image from FIG. 2A.

FIGS. 3A-3B illustrate embodiments of an image of a mobile device cameraview of a live scene displayed on the screen of a mobile device while anauxiliary object is docked to a feature visible in the image.

FIGS. 4A-4D illustrate embodiments of an image of a mobile device cameraview of a live scene displayed on the screen of a mobile device while anauxiliary object is docked to a feature in the image.

FIG. 5A illustrates an example of a method for displaying, in a mobiledevice, an augmented image.

FIG. 5B illustrates an example of a method for displaying an image of amobile device camera view of a live scene while displaying at least oneauxiliary object.

FIG. 6 illustrates an example of a mobile device according toembodiments of the disclosure.

DETAILED DESCRIPTION

Some example techniques are presented herein which may be implemented invarious methods, means and apparatuses in a mobile device and/or in anaugmented reality and/or in a navigation-related system to provide foror otherwise support the ability to display multiple kinds ofinformation and/or data to a user.

Example techniques presented herein address various methods andapparatuses in a mobile device to possibly provide for or otherwisesupport displaying a live scene simultaneously with at least onenon-live-scene, or auxiliary, object. Navigation applications are oneexample of applications that could be improved by simultaneous displayof or gesture-based alternate display of live scene with an auxiliaryobject such as, in one example, a top-down map associated with thelocation of a mobile device. In an augmented reality context, the livescene can additionally be augmented by augmented reality data.

FIG. 1A illustrates a user navigating a venue 100. As illustrated, thevenue 100 is a mall, but it is understood that the venue can include anyindoor or outdoor venue. More generally, the venue can include anylocation at which a user is using the mobile device and the surroundingsof the location. As illustrated, a hand 110 of a user is holding amobile device 112. The mobile device 112 includes a screen 114 fordisplay of images, data, and/or information to the user. The user mayuse the mobile device 112 to navigate within the venue 100, and,additionally or alternatively, the user may use the mobile device 112 toaccess data about the environment. While navigating within the venue100, the user may use a top-down map to understand the user's locationwithin the venue and, for example, to navigate to a particular locationwithin the venue or to otherwise learn more about the venue. Such a mapis illustrated in FIG. 1B.

With continued reference to FIG. 1A, FIG. 1B illustrates a map 140showing a location of the user inside the venue 100. The map 140indicates the location of the user by displaying a location indicator142, here illustrated as a dot pin drop. Map 140 may, in variousimplementations, be displayed on the screen 114 of the mobile device 112to help user navigate within the venue 100. In the illustrated example,the map 140 is a top-down map and includes a route 144 to a destinationlocation 146, here illustrated as the “KidsZone” location. The route 144could be further augmented with text-based guidance 148 or audio-basedguidance to navigate to the destination location 146. The map 140 mayalso illustrate or otherwise highlight points of interest along theroute 144, traffic conditions (including pedestrian traffic conditions),and/or areas of slow traffic. The mobile device 112 may also receiveestimated time to traverse the route or parts of the route thereof,estimated route distance, and other route related information. In anembodiment, the information may be updated as conditions change. In anembodiment, the updated received information may be sent along withongoing navigation instruction updates.

Although not illustrated, it is also understood that the map 140 may beaugmented with data related to locations in the venue 100 or usersurroundings. In the illustrated example, one or more of the illustratedmall locations, “5 Dollars or Less,” “All About Hair,” “Sunglasses +,”“Cold Cream,” “Games Plus,” “Res Publica Coffee,” and “KidsZone,” couldalso be associated with augmented reality data such as a displayed tagindicating a category (such as a box or other tag indicating the kind ofstore it is, for example, “variety store,” “department store,”“off-price department store,” “accessories,” “food,” “games/hobbies,”“indoor playground,” etc.), information about goods or services sold byeach of the one or more mall locations, and/or marketing informationabout promotions or sales currently in effect at each of the one or morestores.

With continued reference to FIGS. 1A and 1B, FIG. 1C illustrates animage of mobile device camera view 160 corresponding to an image of alive scene displayed on the screen 114 of a mobile device 112 where theimage of mobile device camera view 160 of the live scene is augmentedwith augmented reality (AR) data 162 a, 162 b, and 162 c. As usedherein, AR is a live direct or indirect view of a physical, real-worldenvironment whose elements are “augmented” by computer-generated sensoryinput such as sound, video, text and/or graphics, or other data. Asillustrated, the AR data is displayed on an image of a mobile devicecamera view (the image of a mobile device camera view corresponding toan image of the live scene captured by a camera of the mobile device112) and displayed on the screen 114 in real time. As used herein, “livescene” is intended to denote an image or series of images (for example,a video) of a mobile device camera view of the environment around themobile device in real time. Augmentation techniques used for displayingAR data 162 a-162 c are typically performed in real time and usesemantic context with environmental elements, such as overlaying AR data162 a-162 c near or at real-world objects, such as the signage for malllocations “5 Dollars or Less,” “All About Hair,” and “Cold Cream,” basedon recognition of the real-world objects and/or mobile device location.AR can be used to integrate marketing with the image of mobile devicecamera view 160 to generate an augmented image. Real-world objects,environmental elements, and/or semantic context recognized in images ofa live scene, for example, as illustrated, the signage 164 a, 164 b, 164c at mall locations bearing the name of the locations, can providereal-world context and server as a “trigger” for an AR system to displayAR data 162 a-162 c. In other implementations, real world imagescontaining machine readable data, such as a two dimension bar code, forexample a quick response (QR) code, can serve as such a trigger.

The AR data 162 a-162 c can include information related to thelocations. For example, the AR data 162 a-162 c can include informationabout promotions or sales at a particular location. The AR data 162a-162 c can include tags, for example, determined based on thedetermined location of the mobile device 112 or based on featuresidentified in the captured image of the live scene. The AR data 162a-162 c can include links, for example, a link to a website associatedwith the location, a link to an introductory video about the goods andservices sold at the location, or a link to a coupon (such as AR data162 c) or other promotional material. AR data 162 a-162 c can also allowa customer or prospective customer to view the interior of the location(“interior” here referring to interior relative to the illustratedcorridor 166 of the mall), either by providing a link that can cause themobile device 112 to display a new screen with images or video of theinside of the store, or additionally or alternatively, by showing suchimages or video within one of the illustrated call-out bubbles fordisplaying AR data 162 a-162 c, or other appropriate method fordisplaying such video along with, or alternative to, the image of mobiledevice camera view 160. In some implementations, the AR data 162 a-162 cis user-specific and targeted to a particular user based on other datathat the system has about the user, such as a user profile, userpreferences or favorites, and/or past purchases. For example, based onthe date of the last purchase of conditioner for a particular user, andhistorical data about conditioner purchases, AR data 162 b reminds theuser that the user may need to buy conditioner, which is sold at the“All About Hair” mall location.

In the context of the image of mobile device camera view 160 enhancedwith AR data 162 a-162 c, the mobile device 112 may be capable of,either independently or in conjunction with a server, identifyingenvironment elements or other real-world-related features (where thefeatures correspond to areas within the image) within the image ofmobile device camera view 160 that have, for example, a semantic meaninguseful to the user or do not have a semantic meaning but are otherwisefeatures distinguishable from other features within the image. Forexample, within the image of the mobile device camera view 160illustrated in FIG. 1C, feature 168 a can be identified as a floor.Techniques such as ground homography can identify, within an image, aground plane which can be used to identify the floor. Using othermachine vision techniques, such as ridge detection, edge detection,and/or blob analysis and other machine vision techniques, otherenvironmental elements within the image may also be identified as“features” that are distinct from other features. For example, feature168 b can be identified as a passage way through which the corridor 166enters into a new corridor and/or corridor 166 continues to the leftfrom said passage. In the illustrated embodiment, the passage way formsa quadrilateral associated with the feature 168 b corresponding to thepassage way. Similarly, feature 168 c can be recognized as a ceiling andfeatures 168 d and 168 e can be recognized as walls. Recognition ofenvironmental elements which define features can be machine-based by anaugmented reality system or, additionally or alternatively, they may beuser-defined. Although the illustrated features 168 a-168 e happen tocorrespond to polygons or polygon-like shapes, it is understood that anfeature within the image of the camera view of the live scene may beirregular (although, it is also understood that even irregular shapescan be approximated as polygons of a sufficient number of sides). Forexample, an feature could comprise a winding corridor or road, or apuddle, lake, or other body of water to name but a few non-limitingexamples.

Since mobile device 112 users can benefit from simultaneous and/oralternative display of map data as illustrated in FIG. 1B and AR data asillustrated in FIG. 1C, methods of allowing for simultaneous oralternative display of such information can provide for improved userexperience.

FIG. 2A illustrates an image of a mobile device camera view 210 of alive scene displayed on the screen of a mobile device (similar to screen114 of mobile device 112 of FIG. 1A) where the image is augmented withaugmented reality data while an auxiliary object is docked to a featurevisible in the image. FIG. 2B illustrates an image of a mobile devicecamera view 220 of a live scene similar to image of mobile device cameraview 210 illustrated in FIG. 2A where the auxiliary object is docked toa different feature in the image from FIG. 2A. With continued referenceto both FIGS. 2A and 2B, as illustrated, the auxiliary object comprisesa map 240 similar to the map 140 of FIG. 1B. For example, the map 240can be top-down map associated with the live scene (and is alsoassociated with a location of the mobile device). Similarly, asillustrated in FIG. 1C, the image is augmented with AR data 262 a, 262b, and 262 c similar to AR data 162 a-162 c. As illustrated in FIG. 2A,the map 240 is docked to a feature 268 a corresponding to anenvironmental element corresponding to a floor in the image, while inFIG. 2B, the map 240 is docked to a feature 268 b corresponding to anenvironmental element corresponding to a passage way at the end of thecorridor 266. Although the environmental elements correspond toreal-world/physical features such as physical objects, things, orarchitectural features in the live scene, from the perspective of theimage of the camera view of the live scene displayed on the screen ofthe mobile device, the environmental element is simply an area withinthe image. The auxiliary object, in this example the map 240, cantherefore be displayed “docked” to the area of the live scenecorresponding to the, for example, physical object. Examples of physicalobjects corresponding to a feature visible in a captured image to whichan auxiliary object may be docked can include a wall, a corner, a floor,an edge (such as an edge between a floor and a wall), a storefront, awindow or other physical and/or architectural object, or a combinationthereof. It is understood, however, that the “feature” visible in theimage need not necessarily correspond to a physical object, but maycorrespond to a collection of physical objects, or to an arbitrary shapewithin the image. In one aspect of the auxiliary object being docked toa feature visible in the image, the auxiliary object covers a portion ofthe image associated with the feature. It is understood that the objectmay be docked to a feature visible in the captured image, but that suchfeature may not be, in some implementations, visible in the augmentedimage as the feature may be obscured by the auxiliary object.

The map 240 being docked to the a feature (such as a features 268 a and268 b) visible in the captured image can be understood to mean that asthe augmented image of the mobile device camera view of the live sceneis updated to be displayed in real-time on the display of the mobiledevice, the auxiliary object remains docked to the feature of thecaptured image as the mobile device camera view changes as, for example,the mobile device is moved. For example, an orientation of the mobiledevice may change such that more or less of the area in the capturedimage corresponding to the feature to which the map 240 is docked showsin the captured image of the mobile device camera view, and therefore,more or less, as the case may be, of the map 240 can be displayed insubsequent displayed augmented images of the mobile device camera view.In the case of a feature that does not correspond to a physical object,for example, an arbitrarily defined area within the captured image, themobile device, or a processor within the mobile device, may then anchorthe arbitrarily defined area to keypoints within the captured image(corners, lines, shapes, etc.) so that as the mobile device moves aboutwithin the environment and due to geometric translation and/or rotationof the mobile device, the auxiliary object remains docked to thearbitrarily defined area, coming into and out of view as the keypointswithin the captured image come into and out of view due to changes inthe position and orientation of the mobile device.

Referring back to FIG. 1C, in image of mobile device camera view 160, aplurality of features 168 a-168 e were identified. In oneimplementation, a user interacting with the image of mobile devicecamera view 160 may select one of the features for display of the map240. As such, the mobile device, a processor within the mobile device,and/or an input device of the mobile device (such as a touchscreen, forexample, which may, in one example, be overlaid over a display/screen ofthe mobile device), may receive an input designating the feature or areaof the captured image for docking the at least one auxiliary object. Asillustrated in FIGS. 2A and 2B, the user selected, for example, features168 a and 168 b in FIG. 1C, respectively, for display of the map 240.However, in some implementations the auxiliary object could beautomatically docked to a default location or feature which may or maynot then be changed by the user. Although the map has been referredherein as an auxiliary object, it is understood that, in someimplementations, it may also be considered itself a type of AR datasince it is an image (the image being a kind of data) that is notphysically in the live scene but is added to the captured image of thecamera view of the live scene by the mobile device to generate anaugmented image. Although auxiliary object can be a type of AR data, itis being referred to here as a “auxiliary object” to distinguish it fromother kinds of AR data, such as AR data 262 a-262 c. Hence, asillustrated in FIGS. 2A and 2B, the mobile device may display the imageof the mobile device camera view with auxiliary object docked tofeatures 268 a, 268 b while further displaying, in addition to the atleast one auxiliary object, system generated AR data 262 a-262 c.

Map 240 shown in FIG. 2A can be displayed and oriented in differentways. For example, in one implementation, the map can be independentlyscalable while docked to the feature 268 a of the image of the mobiledevice camera view 210 of the live scene. Additionally or alternatively,the map may be enlarged, reduced, panned, translated, or orientedindependently of the remainder of the augmented image. In such anembodiment, the feature 268 a, in this example the floor, serves as alocation within the captured image where the map can be displayed. Thismay be useful, for example, so that features within the captured imagethat are of less interest to the user are used to display the map,allowing the remainder of the image for other kinds of AR data 262 a-262c annotation. As illustrated, the map 240 matches the actual topology ofthe corridor in the live scene and includes a navigation path 244indicating a route to a destination, for example destination 146 withreference to FIG. 1B. However, it is understood that the auxiliaryobject docked to, in this example, the floor, can include a map thatdoes not match the actual topology. For example, the map could be a mapof the entire venue without matching locations in the map with visibleobjects in the live scene. Alternatively, the map docked to the floorcould be a map of a different location than the venue in which themobile device is located. As described elsewhere herein, the auxiliaryobject need not be a map at all, and can include a browser, a userinterface for any application on the mobile device (such as a textingapplication, a telephone application, etc.), a display area for an imageor a video, etc.

Although FIGS. 2A and 2B show embodiments in a corridor of a mall, it isunderstood that the camera view image may also include the inside of astore, for example one of mall locations “5 Dollars or Less,” “All AboutHair,” “Sunglasses +,” etc. For example, in FIG. 2A with the map 240docked to the floor, when entering a store, a map of the inside of thestore can be displayed in the area of the floor. AR data related toitems within the store, similar to AR data 262 a-262 c, can displayed incoordination with the map 240. In one unillustrated example, instead ofthe call-out balloons illustrated in FIG. 2A, the AR data 262 a-262 ccan include display or signpost icons within which information, such astags indicating the merchandise displayed in different areas of thestore or other information of interest and/or marketing information forshoppers within the store, can be displayed or written. Such displays orsignposts could, in one example, rise out of the floor as userapproaches a region within the store. In one implementation, displaysand signposts are associated with the map 240 that is displayed on thefloor and/or corridors on the floor.

FIGS. 3A-3B illustrate embodiments of an image of a mobile device cameraof a live scene displayed on the screen of a mobile device while anauxiliary object is docked to a feature visible in the image. In thisexample, FIG. 3A illustrates an image of a mobile device camera view 310similar to that of FIG. 2A, while 3B illustrates an image of the mobiledevice camera view 320 where the mobile device has been moved and/orrotated so that the floor of the venue covers more of the screen of themobile device as compared to FIG. 3A. Due to the change in the field ofview of the captured image of the mobile device camera view, thephysical signage “5 Dollars or Less” is no longer visible, and hence, ARdata now includes a tag 362 a′ indicating the name of the store as wellas a store category, “5 Dollars or Less, Discount Store.” Furthermore,AR data 362 a continues to be displayed pointing to what remains of theouter wall of the “5 Dollars or Less” location, while in the viewillustrated in FIG. 3A, AR data 362 a pointed to the physical signage.

Notably, feature 368 a′ in FIG. 3B covers much more of the capturedimage of the mobile device camera view 320 displayed on the mobiledevice than feature 368 a in FIG. 3A. As such, map 340 is illustratedfilling more of the augmented image displayed on the mobile device inFIG. 3B compared to FIG. 3A. Hence, as the augmented image of the mobiledevice camera view 320 was updated in real time due to the mobile devicebeing moved, rotated, or re-oriented from the time of the capture anddisplay of the augmented image of the mobile device camera view 310 inFIG. 3A to the time of the capture and display of the augmented image ofthe mobile device camera view 320 in FIG. 3B, the auxiliary object,illustrated here as map 340, remains docked to the features 368 a, 368a′ of the live scene as the features 368 a, 368 a′ increase in size,decrease in size, and/or move and/or come into, and out of, view withinthe captured image. As illustrated, map 340 is shown at a larger scalesince feature 368 a′ in FIG. 3B is larger than 368 a in FIG. 3A.However, in alternative implementations, responsive to the increase inthe size of the feature 368 a within the captured image of the mobiledevice camera view due to movement of the mobile device, the screendisplays a larger portion of the at least one auxiliary object at aconsistent scale as the feature 368 a goes from smaller to larger. Also,as illustrated, although there is a change in orientation and/orattitude of the mobile device from FIG. 3A to FIG. 3B, an orientation ofthe auxiliary object remains consistent relative, in this example, tothe edge between the floor and the wall. In alternative embodiments, theorientation of the auxiliary object, in the illustrated example, map340, could change as the orientation of the mobile device is changed.Furthermore, as illustrated in FIG. 3A, the map 340 displayed on thefloor corresponds to the other actual objects in the image displayed onthe screen. However, as illustrated in FIG. 3B, the map no longercorresponds to the other actual objects in the image, as the map 340still displays the “All About Hair,” “Sunglasses +,” and “Cold Cream”stores even though only the entrance to the “5 Dollars or Less!” storeremains visible in the image.

FIGS. 4A-4D illustrate embodiments of an image of a mobile device cameraview of the live scene displayed on the screen of a mobile device whilean auxiliary object is docked to a feature in the image. In thisexample, FIG. 4A illustrates a situation similar to that of FIG. 2B,while FIGS. 4B-4D illustrate successive images of the mobile devicecamera view 410, 420, 430, 450 where the mobile device is being moveddown the corridor 466, thereby drawing closer to (and eventually, inFIG. 4D, crossing,) the passage (corresponding to features 468 b, 468b′, 468 b″) at the end of corridor 466. As a result, the features 468 b,468 b′, 468 b″ in the image of the mobile device camera view 410, 420,430 successively get larger within the image of the mobile device cameraview. Although in the illustrated example, as features 468 b increasesin size from FIGS. 4A-4C, map 440 is illustrated increasing in scale.However, in alternative implementations, responsive to the increase inthe size of the feature 468 b, 468 b′, 468 b″ within the image of themobile device camera view 410, 420, 430 due to movement of the mobiledevice, the screen displays a larger portion of the at least oneauxiliary object at a consistent scale as the features 468 b, 468 b′,468 b″ go from smaller to larger.

In the example shown in FIGS. 4A-4D, the augmented image of the mobiledevice camera view 410, 420, 430, 450 is updated in real time as themobile device is moved, where the auxiliary object, in the illustratedexample, the map 440, remains docked to the feature 468 b of the mobiledevice camera view as the mobile device is moved. Finally, at FIG. 4D,the user, and by extension the mobile device, has passed through thepassage corresponding to features 468 b, 468 b′, 468 b″. In thisexample, the map 440 appeared in the display as a “wall” upon which themap 440 was projected. Once the user has passed through the “wall,” inFIG. 4D, the map 440 is no longer visible. The map 440 remained dockedto the features 468 b, 468 b′, 468 b″ throughout FIGS. 4A-4C until, atFIG. 4D, responsive to the feature 468 b no longer being within thecaptured image of the mobile device camera view due to movement of themobile device, the display or screen no longer displays the at least oneauxiliary object, in this example, map 440. In such a case, the mobiledevice, or one or more processors within the mobile device, remove theat least one auxiliary object from the augmented image. Now that “ResPublica Coffee” is in view in FIG. 4D, in one example, AR data 462 drelated to this mall location is displayed in the augmented image of themobile device camera view 450. In an alternative implementation notillustrated, once the map is no longer displayed in feature 468 b due tofeature 468 b no longer remaining within the mobile device camera view450, the map may be automatically, by default, docked to another featurein the captured image of the mobile device camera view, for examplefeature 470 corresponding to the floor. In one implementation, forexample, the map (or any other auxiliary object) may, due to feature 468b no longer remaining within the mobile device camera view 450, snap toanother feature, where the other feature is visible in the capturedimage. The other feature can be, for example, a similar feature tofeature 468 b, or could alternatively be a default feature for dockingthe auxiliary object. For example, supposing that feature 468 b were afirst window (or first storefront, etc.), and due to movement of themobile device the first window (or first storefront, etc.) no longerremained within the mobile device camera view 450, the map couldautomatically snap to a second window (or second storefront, etc.).Hence, supposing that an auxiliary object was docked to the entrance tothe “5 Dollars or Less!” store in FIG. 4A to generate the augmentedimage, responsive to that entrance no longer being within the field ofview of the captured image in FIG. 4B, the auxiliary object may snap tothe “Sunglasses +” entrance of FIG. 4B and be displayed in the augmentedimage as docked to the entrance of the “Sunglasses +” store. Finally, inFIG. 4C, where only a portion of the entrance of the “Sunglasses +”store is visible, the auxiliary object could snap to the entrance of the“Cold Cream” store and be displayed in the augmented image as docked tothe entrance of the “Cold Cream” store.

In one aspect of FIGS. 4A-4D, map 440 is also updated as the image ofthe mobile device camera view changes due to mobile device movement. Asshown in FIG. 4A, an indication of a determined location of the mobiledevice is indicated with the pin drop illustrating that the user is nearthe “5 Dollars or Less!” store. As the mobile device moves, and asubsequent augmented image of the mobile device camera view isdisplayed, the map 440 is also updated by updating the indication of thedetermined location of the mobile device. In FIG. 4B, the map 440 isupdated by updating the indication of the determined location of themobile device and hence the pin drop now indicates that the user is nearthe “Sunglasses +” store. Finally, in FIG. 4C, the map 440 is updatedand the pin drop now shows that the user is near the “Cold Cream” store.

As illustrated in FIGS. 4A-4C, map 440 remained docked to the passageway (corresponding to features 468 b, 468 b′, 468 b″) of the mobiledevice camera view as the mobile device moved, where the feature 468 bis stationary in the real world while the mobile device draws closer tothe passage way. In another embodiment similar to that illustrated inFIGS. 4A-4C, map 440 (or other auxiliary object) could be docked to aplane in the corridor that remains a certain distance away from the useras the user moves down the corridor. As the user moves, the map 440remains docked a fixed distance away, but the remainder of the augmentedimage changes as the scene changes due to the motion of the user holdingmobile device, for example, as new stores come into view and old storesgo out of view.

FIG. 5A illustrates an example of a method for displaying, in a mobiledevice, an augmented image, for example the augmented images illustratedin FIGS. 2A, 2B, 3A, 3B, 4A, 4B, and 4C. The method 500A can includeblock 525A with capturing, by a camera of the mobile device, an image ofa live scene. The image of the live scene, for example, can include, asone example, image of live scene displayed on the screen 114 of mobiledevice 112 with reference to FIG. 1A. Means for performing thefunctionality of block 525A can include, but are not limited to,camera(s) 650, bus 605, working memory 635, and/or processor(s) 610 ofFIG. 6 . The method 500A continues at block 530A with displaying theaugmented image based on the captured image. Once the image is capturedin block 525A, the mobile device, or one or more processors in themobile device, may determine (or receive an input designating) a featurewithin the captured image to which to dock the auxiliary object andgenerate an augmented image with the auxiliary object docked to thefeature. The augmented image is generated based on the captured imageand the auxiliary object and the augmented image therefore includes theauxiliary object. In some implementations, the auxiliary object may beopaque or may be semi-transparent. In an opaque implementation, theaugmented image includes both the auxiliary object as well as portionsof the captured image not covered by the auxiliary object. In asemi-transparent implementation, the augmented image may include thecaptured image with the semi-transparent auxiliary object coveringportions of the captured image but with aspects of the captured image inthe covered portions still remaining visible “behind” the auxiliaryobject. Block 530A and block 530B of method 500B of FIG. 5B describedbelow can correspond to each other, and the teachings with reference toone block can apply to the other block. The method 500A continues atblock 540A with updating the augmented image as the mobile device ismoved, wherein the at least one auxiliary object remains docked to thefeature as the augmented image is updated. Block 540A and block 540B ofmethod 500B of FIG. 5B described below can correspond to each other, andthe teachings with reference to one block can apply to the other block.

FIG. 5B illustrates an example of a method for displaying an image of amobile device camera view of a live scene while displaying at least oneauxiliary object, for example the images of the mobile device cameraview illustrated in FIGS. 2A, 2B, 3A, 3B, and 4A-4D. The method 500B fordisplaying, in a mobile device, such as mobile device 112 of FIG. 1A, animage of a mobile device camera view of a live scene captured by acamera of the mobile device can optionally include optional block 510Bwith determining a location of the mobile device. One example of amobile device is illustrated with reference to FIG. 6 . Such animplementation for a mobile device can include components capable ofcomputing a location of the mobile device, such as a location engine.The optionally computed mobile device location can be used in variousaspects of the method discussed herein. Means for performing thefunctionality of optional block 510B can include, but are not limitedto, global navigation satellite system (GNSS) receiver 660,communications subsystem(s) 630, location engine 670, bus 605, storagedevice(s) 625, working memory 635, and/or processor(s) 610 of FIG. 6 .

The method 500B continues at optional block 520B with receiving an inputdesignating the area or feature of the image of the mobile device cameraview of a live scene for docking the at least one auxiliary object. Withreference to FIG. 1C, features 168 a, 168 b, 168 c, 168 d (for example)may be automatically identified by the mobile device or may beuser-specified. In implementations where the features visible in theimage for docking the auxiliary object are automatically or by defaultidentified, the default location could be user-defined. Alternatively,the default feature for docking the auxiliary object could be the lastfeature or a feature similar to the last feature on which the auxiliaryobject was docked. Hence, in an embodiment where a user had previouslyselected a window for docking the auxiliary object, the mobile devicemay select a window visible in the image for docking the auxiliaryobject, which the user may then, for example, manually change. In oneimplementation, the user may touch or otherwise select one of features168 a, 168 b, 168 c, 168 d (for example) for docking the auxiliaryobject to the selected feature of the image of the mobile device cameraview, with the touchscreen of the mobile device generating a signal inresponse thereto. In one such implementation, the user may touch or tapan edge of one of features 168 a, 168 b, 168 c, 168 d, or any pointwithin features 168 a, 168 b, 168 c, 168 d, to designate the feature fordocking. Additionally or alternatively, the user may drag the auxiliaryobject, or an icon representing the auxiliary object, over differentportions the image of the live scene. The mobile device may thenhighlight a feature, for example one of features 168 a, 168 b, 168 c,168 d, over which the auxiliary object is currently hovering as the userdrags the auxiliary feature around. When the user releases from touchingthe touchscreen, the mobile device can receive the released touch as aninput for selecting the feature over which the auxiliary object, orauxiliary object icon, was hovering at the time the user's touch wasreleased. Hence, a touchscreen display may be configured to, or one ormore processors controlling the touchscreen display can be configuredto, to receive a tap gesture or a drag gesture by a user and generatethe input to dock the auxiliary object based on the tap or drag gesture.The processor(s) of the mobile device then receives such a signal andrecognizes the signal as an input to dock the auxiliary object to theselected feature. Then, one or more processors of the mobile device areconfigured to dock, responsive to the input, the auxiliary object to thedesignated feature. In one implementation, the input designating thefeature to dock the auxiliary object may include a snap to edge or snapto corner feature, allowing a user to move the auxiliary object over theimage of the camera view of the live scene, where the mobile device isconfigured to snap edges or corners of the auxiliary object to edges orcorners of various features in the image so that user need not perfectlyplace the auxiliary object over the feature to which it is to be docked.Means for performing the functionality of optional block 520B caninclude, but are not limited to, input device(s) 615 (such as, forexample, touchscreen 617), bus 605, working memory 635, and/orprocessor(s) 610 of FIG. 6 . In embodiments where a user selects afeature in the displayed image of the mobile device camera view fordocking the auxiliary object, the touchscreen may receive an input todock the auxiliary object to the feature of the image of the mobiledevice camera view. The touchscreen may also generate a signalresponsive to the touch selection of the user, and hence the processormay also receive an input, for example the signal from the touchscreen,to dock the auxiliary object. Subsequent to the processor receiving theinput, the processor may then execute instructions to display the atleast one auxiliary object docked to the designated feature.

The method 500B continues at block 530B with displaying the image of themobile device camera view including the at least one auxiliary objectdocked to the feature of the image of the mobile device camera view. Asnoted above, the image of the mobile device camera view can be generatedby first capturing an image of the live scene using a camera of themobile device, which may then be displayed on a display or screen of themobile device. With reference to FIGS. 2A and 2B, the auxiliary objectcan include a map, such as a top-down map, associated with the locationof the mobile device determined at optional block 510B. In one suchexample, the top-down map may be displayed on or over a physical objectin the image of the mobile device camera view. In other implementations,the auxiliary object can include a browser, an interface for amobile-based application (e.g., a keypad for a mobile device nativephone application, a text messaging application, a mobile device newsapplication (which could include images, text, or video), a videodisplaying application, etc.), etc. As illustrated in, for example,FIGS. 3A, 3B, and 4A-4D, the image of the auxiliary object replaces theportion of the image of the mobile device camera view in the areacorresponding to the feature to which the auxiliary object is docked.However, in other unillustrated examples, the auxiliary object may betransparent or semi-transparent, allowing some aspects of the live scenein the area corresponding to the feature to be visible as though theauxiliary object were a semitransparent window through which the livescene in the area can be seen. Means for performing the functionality ofblock 530B can include, but are not limited to, display(s) 620,camera(s) 650, bus 605, working memory 635, and/or processor(s) 610 ofFIG. 6 .

The method 500B continues at block 540B with updating the image of themobile device camera view as the mobile device is moved while theauxiliary object remains docked to the feature of the image. Withreference to FIGS. 3A, 3B, and 4A-4D, as the camera of the mobile devicecaptures new images of the live scene in real time as the user movesthrough the user's environment, the image of the mobile device cameraview is updated on the screen of the mobile device and the auxiliaryobject remains docked to the feature of the image of the mobile devicecamera view. The auxiliary object remains docked as the size, shape,and/or orientation of the feature in the image of the mobile devicecamera view changes due to the movement of the user holding the mobiledevice with the camera that is capturing the image of the mobile devicecamera view. In some implementations, the orientation of the auxiliaryobject may also respond to changes in orientation of the image. Hence,responsive to changes in the image of the mobile device camera view dueto changes in the orientation of the camera of the mobile devicecapturing the image, the mobile device may change the orientation of theimage of the auxiliary object displayed in the image of the mobiledevice camera view. Similarly, as the feature pans into, and out of,view within the image of the mobile device camera view, the auxiliaryobject remains docked to the feature. In one embodiment, where theauxiliary object is a top-down map, the top-down map can include anindication of the determined location (block 510B) of the mobile device,as shown by the pin drop in FIGS. 4A, 4B, and 4C. As shown in theexample images in FIGS. 4A, 4B, and 4C, as the image of the mobiledevice camera view is updated as the mobile device moves, this alsoincludes updating the top-down map, for example, by updating theindication of the determined location of the mobile device as thelocation changes as the mobile device is moved. In FIGS. 4A, 4B, and 4C,this is shown as the pin drop moves from figure to figure. Means forperforming the functionality of block 540B can include, but are notlimited to, display(s) 620, input device(s) 615 (including, for example,camera(s) 650, accelerometer(s), magnetometer(s), gyroscope(s)), bus605, working memory 635, and/or processor(s) 610 of FIG. 6 .

The method 500B optionally continues at optional block 550B withdisplaying system generated augmented reality data associated withobjects in the image of the mobile device camera view. With referenceto, for example, FIGS. 2A and 2B, the image of the mobile device cameraview can include, in addition to the auxiliary object docked to thefeature, other AR data 262 a, 262 b, 262 c. Such AR data can beassociated with the location of the mobile device determined in optionalblock 510B. Such data may additionally or alternatively be associatedwith objects detected by the mobile device within the image of themobile device camera view with or without reference to the location ofthe mobile device. Means for performing the functionality of optionalblock 550B can include, but are not limited to, display(s) 620, bus 605,working memory 635, and/or processor(s) 610 of FIG. 6 .

FIG. 6 illustrates an example of a mobile device according toembodiments of the disclosure, such as mobile device having a screendisplaying the various images described elsewhere throughout thisdisclosure. The mobile device can also be considered a computing device600, or alternatively, computing device 600 may represent some of thecomponents of a mobile device or any other computing device. Examples ofmobile device and/or computing device 600 include, but are not limitedto, video game consoles, tablets, smart phones, laptops, netbooks, orother portable devices. FIG. 6 provides a schematic illustration of oneembodiment of computing device 600 that may perform the variousfunctions described herein, including, for example, method 500A of FIG.5A and/or 500B of FIG. 5B. FIG. 6 is meant only to provide a generalizedillustration of various components, any or all of which may be utilizedas appropriate. FIG. 6 , therefore, broadly illustrates how individualsystem elements may be implemented in a relatively separated orrelatively more integrated manner.

Computing device 600 is shown comprising hardware elements that may beelectrically coupled via a bus 605 (or may otherwise be incommunication, as appropriate). The hardware elements may include one ormore processor(s) 610, including, without limitation, one or moregeneral-purpose processors and/or one or more special-purpose processors(such as digital signal processing chips, graphics accelerationprocessors, and/or the like). The one or more processor(s) 610 arecoupled to memory 635, camera(s) 650, and display(s) 620 via bus 605 andcan be configured to present an augmented image on the display(s) 620based on an image captured by camera(s) 650, the augmented imageincluding at least one auxiliary object docked to a feature comprising aphysical object visible in the captured image and subsequently updatethe augmented image as the mobile device is moved such that the at leastone auxiliary object remains docked to the feature as the augmentedimage is updated. The hardware elements may further include one or moreinput devices 615, which may include, without limitation, a touchscreen617, a mouse, a keyboard and/or the like; and one or more display(s)620, which may be configured, for example, to display an image of amobile device camera view, to display AR data, and/or to display anaugmented image based on a captured image of a live scene and anauxiliary object docked to a feature visible in the captured image).While touchscreen 617 and display 620 are illustrated as separate blocksin FIG. 6 , it is understood that they may be integrated into a singledevice as a touchscreen display capable of displaying images and/orvideo while also receiving touch input from a user. Sensor(s) 680 mayinclude camera(s) 650, although illustrated as separate. Camera(s) 650may capture successive frames of images of an environment, therebycapturing images of a camera view of a live scene in real time asdescribed elsewhere in the specification. Sensor(s) 680 may also includemotion sensor(s) 685. Motion sensor(s) 685 can include, for example,accelerometer(s) (for example, a three-axis accelerometer),magnetometer(s), gyroscope(s) (for example, a three-axis gyroscope),which can be used, in some embodiments, to help determine a position,orientation, and/or attitude of the mobile device. For example, once agood position is determined using wireless signals, such as satellitesignals, wide area network signals, or wireless local area networksignals, subsequent positions can be computed using dead reckoningmethods with such sensors for some time before errors eventually becometoo large.

Computing device 600 may further include (and/or be in communicationwith) one or more non-transitory storage device(s) 625, which maycomprise, without limitation, local and/or network accessible storage,and/or may include, without limitation, a disk drive, a drive array, anoptical storage device, a solid-form storage device such as arandom-access memory (RAM) and/or a read-only memory (ROM), which may beprogrammable, flash-updateable and/or the like. Such storage devices maybe configured to implement any appropriate data storage, including,without limitation, various file systems, database structures, and/orthe like.

Computing device 600 may also include a communications subsystem(s) 630.Communications subsystem(s) 630 may include a transceiver for receivingand transmitting data or a wired and/or wireless medium. Communicationssubsystem(s) 630 may also include, without limitation, a modem, anetwork card (wireless or wired), an infrared communication device, awireless transceiver and/or communication device and/or chipset (such asa Bluetooth□ device, a wireless local area network (WLAN) device, an802.11 device, a Wi-Fi device, a wireless wide area network (WWAN)device, a worldwide interoperability for microwave access (WiMax)device, cellular communication facilities, etc.), and/or the like.Communications subsystem(s) 630 may permit data to be exchanged with anetwork, other computing devices, and/or any other devices describedherein. In one example, the communications subsystem(s) 630 may be usedto transmit requests for data, and receive data, such as, for example,map associated with a location of the mobile device, AR data associatedwith the location of the mobile device and/or associated withenvironmental elements and/or objects in an image a camera view of alive scene associated with the location of the mobile device. Moregenerally, data associated with an auxiliary object, such as an image ofthe auxiliary object, data to generate such an image of the auxiliaryobject or other data associated with the auxiliary object may also bereceived using the communications subsystem(s) 630. In oneimplementation, the communications subsystem(s) 630 can be configured tocommunicate with one or more servers, for example, to transmit requestsfor data to the one or more servers associated with the auxiliary object(or augmented reality data) in order to display the auxiliary object (oraugmented reality data), and to receive the data from the one or moreservers and communicate the received data to the processor(s) 610 forfurther processing in order to, for example, display the auxiliaryobject in the augmented image. In one example, the communicationssubsystem(s) 630 transmits a location determined for the mobile deviceto a map server, receives data for displaying the map from the mapserver, and communicates the received data to, for example, processor(s)610 for displaying the map docked to the feature on the display(s) 620.In implementations where the map will also be displaying a route fornavigating or walking to a location, the communications subsystem(s) 630may also request route data from a route server to enable the mobiledevice to further display a route on the displayed map. In otherexamples, the data associated with the auxiliary object or augmentedreality data can include a video feed from a remote camera, data fromservers associated with the display of augmented reality data (such asdata from marketing servers, or servers owned by a business within thevenue), data from servers that deliver data associated with a browser(where the auxiliary object is a browser window), or data associatedwith a mobile device application where the mobile device applicationreceives such data from a remote server. Additionally, inimplementations where communications subsystem(s) 630 include wirelesstransceiver(s), such wireless transceiver(s) can help determine thelocation of the mobile device by performing measurements of wirelesssignals, such as WLAN and WWAN signals.

Computing device 600 may also include a global navigation satellitesystem (GNSS) receiver 660 and GNSS antenna 662. The GNSS receiver 660may measure various signals received from satellites belonging to asatellite positioning system (SPS) such as global positioning system(GPS), GLObal NAvigation Satellite System (GLONASS), Galileo or Beidou.These signal measurements may be utilized to determination a location ofthe mobile device either alone or in combination with terrestrialsignals such as WAN, WLAN and/or other wireless signals which may bereceived and/or measured by, for example, communications subsystem(s)630. In one optional example, the computing device 600 may also includea location engine 670 which can reside within working memory 635,communications subsystem(s) 630, GNSS receiver 660, and/or elsewhere,which may be configured to take inputs from these various subsystemsincluding, but not limited to, for example, GNSS receiver 660,communications subsystem(s) 630, input device(s) 615 such as sensors (inparticular, motion sensors), to compute a location of the mobile device.

In many embodiments, computing device 600 may further comprise anon-transitory working memory 635, which may include a read-only memory(ROM) device. Computing device 600 may comprise software elements, shownas being currently located within the working memory 635, including anoperating system 640, device drivers, executable libraries, and/or othercode, such as one or more application program(s) 645, which may comprisecomputer programs provided by various embodiments, and/or may bedesigned to implement methods, and/or configure systems, provided byother embodiments, as described herein. An interface for one or moreapplication program(s) 645 could be an auxiliary object that is dockedto a feature of a mobile device camera view displayed on a screen of amobile device. Additionally or alternatively, an application programcould function to display the image of the mobile device camera viewwith the auxiliary object docked to the feature of the image. Merely byway of example, one or more procedures described with respect to themethod(s) discussed above might be implemented as code and/orinstructions executable by a computer (and/or a processor within acomputer), such as the example methods illustrated in FIGS. 5A and/or5B; in an aspect, then, such code and/or instructions may be used toconfigure and/or adapt a general purpose computer (or other device) toperform one or more operations in accordance with the described methods.

As described elsewhere in greater detail, such instructions stored on anon-transitory computer readable medium for execution by one or moreprocessors of a mobile device may, in various implementations, includeinstructions to perform operations corresponding to any of the functionsdisclosed herein, for example those illustrated in FIGS. 5A and/or 5B,the operations comprising, for example, capturing, using camera(s) 650,an image of a live scene, displaying, on display(s) 620, an augmentedimage based on the captured image, the augmented image including atleast one auxiliary object docked to a feature comprising a physicalobject visible in the captured image, and updating the augmented imageon display(s) 620 as the mobile device is moved, wherein the auxiliaryobject remains docked to the feature as the augmented image is updated.The operations may further comprise processing an input designating thefeature of the image for docking the at least one auxiliary objectand/or displaying the at least one auxiliary object docked to thedesignated feature. The operations can also include displaying, on thedisplay of the mobile device, system generated augmented reality data.

A set of these instructions and/or code might be stored on acomputer-readable storage medium, such as storage device(s) 625described above. In some cases, the storage medium might be incorporatedwithin a computing device, such as computing device 600. In otherembodiments, the storage medium might be separate from a computingdevice (e.g., a removable medium, such as a compact disc), and/orprovided in an installation package, such that the storage medium may beused to program, configure and/or adapt a general purpose computer withthe instructions/code stored thereon. These instructions might take theform of executable code, which is executable by computing device 600and/or might take the form of source and/or installable code, which,upon compilation and/or installation on computing device 600 (e.g.,using any of a variety of generally available compilers, installationprograms, compression/decompression utilities, etc.) then takes the formof executable code.

Some embodiments may employ a computing device (such as computing device600) to perform methods in accordance with the disclosure. For example,some or all of the procedures of the described methods may be performedby computing device 600 in response to processor(s) 610 executing one ormore sequences of one or more instructions (which might be incorporatedinto operating system 640 and/or other code, such as an applicationprogram 645) contained in working memory 635. Such instructions may beread into working memory 635 from another computer-readable medium, suchas one or more of storage device(s) 625. Merely by way of example,execution of the sequences of instructions/code contained in workingmemory 635 might instruct processor(s) 610 to perform one or moreprocedures of the methods described herein, for example the methodsillustrated in FIGS. 5A and/or 5B. Similarly, working memory 635 may,together with processor(s) 610 be configured to perform one or moreblocks from the methods illustrated in FIGS. 5 A and/or 5B. Additionallyor alternatively, the instructions contained in the working memory 635may instruct the processor(s) 610 to perform, and/or the working memory635 together with processor(s) 610 may be configured to perform, any ofthe functionality described in any of FIGS. 2A, 2B, 3A, 3B, and 4A-4D.It is understood that processor(s) 610 performing the functionalityassociated with any of FIGS. 2A, 2B, 3A, 3B, 4A-4D, 5A, and 5B mayinclude one or more main processors working in coordination withgraphics processor(s) and/or other processor(s) associated withdisplay(s) 620.

With reference to the appended figures, components that can includememory can include non-transitory machine-readable media. The terms“machine-readable medium” and “computer-readable medium,” as usedherein, refer to any storage medium that participates in providing datathat causes a machine to operate in a specific fashion. In an embodimentimplemented using computing device 600, various computer-readable mediamight be involved in providing instructions/code to processor(s) 610 forexecution and/or might be used to store and/or carry suchinstructions/code (e.g., as signals). Common forms of physical and/ortangible computer-readable media include, for example, a floppy disk, aflexible disk, hard disk, magnetic tape, or any other magnetic medium, aCD-ROM, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, a RAM, a programmable ROM(PROM), an erasable PROM (EPROM), an electrically erasable PROM(EEPROM), a FLASH-EPROM, any other memory chip or cartridge, a carrierwave as described hereinafter, or any other medium from which a computermay read instructions and/or code. In many implementations, acomputer-readable medium is a physical and/or tangible storage medium.Such a medium may take many forms, including, but not limited to,non-volatile media, volatile media, and transmission media. Non-volatilemedia include, for example, optical and/or magnetic disks, such asstorage device(s) 625. Volatile media include, without limitation,dynamic memory, such as working memory 635. Transmission media include,without limitation, coaxial cables, copper wire and fiber optics,including the wires comprising the bus 605, as well as the variouscomponents of communications subsystem 630 (and/or the media by whichcommunications subsystem 630 provides communication with other devices,including air for wireless devices). Hence, transmission media may alsotake the form of waves (including, without limitation, radio, acousticand/or light waves, such as those generated during radio-wave andinfrared data communications). In an alternate embodiment, event-drivencomponents and devices, such as cameras, may be used, where some of theprocessing may be performed in the analog domain.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to processor(s) 610 forexecution. Merely by way of example, the instructions may initially becarried on a magnetic disk and/or optical disc of a remote computer. Aremote computer might load the instructions into its dynamic memory andsend the instructions as signals over a transmission medium to bereceived and/or executed by the computing device 600. These signals,which might be in the form of electromagnetic signals, acoustic signals,optical signals and/or the like, are all examples of carrier waves onwhich instructions may be encoded, in accordance with variousembodiments of the disclosure.

Communications subsystem 630 (and/or components thereof) generally willreceive the signals, and bus 605 then might carry the signals (and/orthe data, instructions, etc., carried by the signals) to working memory635, from which processor(s) 610 retrieves and executes theinstructions. The instructions received by working memory 635 mayoptionally be stored on non-transitory storage device(s) 625 eitherbefore or after execution by processor(s) 610.

The methods, systems, and devices discussed above are examples. Variousembodiments may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods described may be performed in an order different from thatdescribed, and/or various stages may be added, omitted, and/or combined.Also, features described with respect to certain embodiments may becombined in various other embodiments. Different aspects and elements ofthe embodiments may be combined in a similar manner. Also, technologyevolves and, thus, many of the elements are examples that do not limitthe scope of the disclosure to those specific examples.

Specific details are given in the description to provide a thoroughunderstanding of the embodiments. However, embodiments may be practicedwithout these specific details. For example, well-known circuits,processes, algorithms, structures, and techniques have been shownwithout unnecessary detail in order to avoid obscuring the embodiments.This description provides example embodiments only, and is not intendedto limit the scope, applicability, or configuration of the disclosure.Rather, the preceding description of the embodiments will provide thoseskilled in the art with an enabling description for implementingembodiments of the disclosure. Various changes may be made in thefunction and arrangement of elements without departing from the spiritand scope of the disclosure.

Also, some embodiments were described as processes depicted as flowdiagrams or block diagrams. Although each may describe the operations asa sequential process, many of the operations may be performed inparallel or concurrently. In addition, the order of the operations maybe rearranged. A process may have additional steps not included in thefigure. Furthermore, embodiments of the methods may be implemented byhardware, software, firmware, middleware, microcode, hardwaredescription languages, or any combination thereof. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the associated tasks may be stored in acomputer-readable medium such as a storage medium. Processors mayperform the associated tasks.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

Those of skill in the art will appreciate that information and signalsused to communicate the messages described herein may be representedusing any of a variety of different technologies and techniques. Forexample, data, instructions, commands, information, signals, bits,symbols, and chips that may be referenced throughout the abovedescription may be represented by voltages, currents, electromagneticwaves, magnetic fields or particles, optical fields or particles, or anycombination thereof.

Terms, “and” and “or” as used herein, may include a variety of meaningsthat also is expected to depend at least in part upon the context inwhich such terms are used. Typically, “or” if used to associate a list,such as A, B, or C, is intended to mean A, B, and C, here used in theinclusive sense, as well as A, B, or C, here used in the exclusivesense. In addition, the term “one or more” as used herein may be used todescribe any feature, structure, or characteristic in the singular ormay be used to describe some combination of features, structures, orcharacteristics. However, it should be noted that this is merely anillustrative example and claimed subject matter is not limited to thisexample. Furthermore, the term “at least one of” if used to associate alist, such as A, B, or C, can be interpreted to mean any combination ofA, B, and/or C, such as A, AB, AA, AAB, AABBCCC, etc.

Having described several embodiments, various modifications, alternativeconstructions, and equivalents may be used without departing from thespirit of the disclosure. For example, the above elements may merely bea component of a larger system, wherein other rules may take precedenceover or otherwise modify the application of the embodiments describedherein. Also, a number of steps may be undertaken before, during, orafter the above elements are considered. Accordingly, the abovedescription does not limit the scope of the disclosure.

In view of this description embodiments may include differentcombinations of features. Implementation examples are described in thefollowing numbered clauses:

Clause 1: A method for displaying, in a mobile device, an augmentedimage, the method comprising: capturing, by a camera of the mobiledevice, an image of a live scene; determining a location of the mobiledevice; and displaying, in a display of the mobile device, the augmentedimage based on the captured image, the augmented image comprising:traffic conditions, estimated time to a destination, route-relatedinformation, or any combination thereof; at least a portion of the imageof the live scene; and a map configured to indicate one or morenavigation instructions toward the destination.

Clause 2: The method of clause 1, wherein the map is independentlyadjustable to the live scene, wherein independently adjustable comprisesenlarged, reduced, panned, translated, oriented, or any combinationthereof.

Clause 3: The method of any of clauses 1-2 wherein the map is docked toone or more features of the live scene, and the one or more features areuser-defined, user-selected, or any combination thereof.

Clause 4: The method of any of clauses 1-3 the method further comprisingreceiving user input indicating the one or more features to use fordocking the map.

Clause 5: The method of any of clauses 1-4 the method further comprisingoverlaying the map in the live scene over an area associated with theone or more features, the map comprising a navigation path indicating aroute to the destination, the one or more navigation instructionscomprising the navigation path.

Clause 6: The method of any of clauses 1-5 the method further comprisingobtaining one or more augmented reality (AR) data associated with thelocation of the mobile device; and displaying the one or more AR data inthe augmented image based on one or more second features detected in thelive scene.

Clause 7: A mobile device comprising: a camera configured to capture animage of a live scene; a display; a memory; a location engine configuredto determine a location of the mobile device; and one or more processorscommunicatively coupled to the camera, the display, and the memory, theone or more processors configured to: capture, by the camera of themobile device, the image of the live scene; and display, in the displayof the mobile device, an augmented image based on the captured image,the augmented image comprising: traffic conditions, estimated time to adestination, route-related information, or any combination thereof; atleast a portion of the image of the live scene; and a map configured toindicate one or more navigation instructions toward the destination.

Clause 8: The mobile device of clause 7, wherein the map isindependently adjustable to the live scene, wherein independentlyadjustable comprises enlarged, reduced, panned, translated, oriented, orany combination thereof.

Clause 9: The mobile device of any of clauses 7-8 wherein the map isdocked to one or more features of the live scene, and the one or morefeatures are user-defined, user-selected, or any combination thereof.

Clause 10: The mobile device of any of clauses 7-9 wherein the one ormore processors are further configured to receive user input indicatingthe one or more features to use for docking the map.

Clause 11: The mobile device of any of clauses 7-10 wherein the map isoverlaid in the live scene over an area associated with the one or morefeatures.

Clause 12: The mobile device of any of clauses 7-11 the one or moreprocessors are further configured to obtain, via one or moretransceivers, one or more augmented reality (AR) data associated withthe location of the mobile device; and display, via the display, the oneor more AR data in the augmented image based on one or more secondfeatures detected in the live scene.

Clause 13: A mobile device comprising: means for capturing an image of alive scene; means for determining a location of the mobile device; andmeans for displaying an augmented image based on the captured image,wherein the augmented image comprises: traffic conditions, estimatedtime to a destination, route-related information, or any combinationthereof; at least a portion of the image of the live scene; and a mapconfigured to indicate one or more navigation instructions toward thedestination.

Clause 14: The mobile device of clause 13, wherein the map isindependently adjustable to the live scene, wherein independentlyadjustable comprises enlarged, reduced, panned, translated, oriented, orany combination thereof.

Clause 15: The mobile device of any of clauses 13-14 wherein the map isdocked to one or more features of the live scene, and the one or morefeatures are user-defined, user-selected, or any combination thereof.

Clause 16: The mobile device of any of clauses 13-15 the mobile devicefurther comprising means for receiving user input indicating the one ormore features to use for docking the map.

Clause 17: A non-transitory computer-readable medium for displaying anaugmented image, the non-transitory computer-readable medium comprisingprocessor-executable program code configured to cause one or moreprocessors to: capture an image of a live scene; determine a location ofa mobile device; and display the augmented image based on the capturedimage, wherein the augmented image comprises: traffic conditions,estimated time to a destination, route-related information, or anycombination thereof; at least a portion of the image of the live scene;and a map configured to indicate one or more navigation instructionstoward the destination.

Clause 18: The non-transitory computer-readable medium of clause 17,wherein the map is independently adjustable to the live scene, whereinindependently adjustable comprises enlarged, reduced, panned,translated, oriented, or any combination thereof.

Clause 19: The non-transitory computer-readable medium of any of clauses17-18 wherein the map is docked to one or more features of the livescene, and the one or more features are user-defined, user-selected, orany combination thereof.

Clause 20: The non-transitory computer-readable medium of any of clauses17-19 wherein the processor-executable program code is furtherconfigured to cause the one or more processors to: obtain one or moreaugmented reality (AR) data associated with the location of the mobiledevice; and display the one or more AR data in the augmented image basedon one or more second features detected in the live scene.

What is claimed is:
 1. A method for displaying, in a mobile device, an augmented image, the method comprising: capturing, by a camera of the mobile device, an image of a live scene; determining a location of the mobile device; and displaying, in a display of the mobile device, the augmented image based on the captured image, the augmented image comprising: traffic conditions, estimated time to a destination, route-related information, or any combination thereof; at least a portion of the image of the live scene; and a map configured to indicate one or more navigation instructions toward the destination, wherein the map is docked to one or more features of the live scene.
 2. The method of claim 1, wherein the map is independently adjustable to the live scene, wherein independently adjustable comprises enlarged, reduced, panned, translated, oriented, or any combination thereof.
 3. The method of claim 1, wherein the one or more features are user-defined, user-selected, or any combination thereof.
 4. The method of claim 3, the method further comprising receiving user input indicating the one or more features to use for docking the map.
 5. The method of claim 3, the method further comprising overlaying the map in the live scene over an area associated with the one or more features, the map comprising a navigation path indicating a route to the destination, the one or more navigation instructions comprising the navigation path.
 6. The method of claim 1, the method further comprising: obtaining one or more augmented reality (AR) data associated with the location of the mobile device; and displaying the one or more AR data in the augmented image based on one or more second features detected in the live scene.
 7. A mobile device comprising: a camera configured to capture an image of a live scene; a display; a memory; a location engine configured to determine a location of the mobile device; and one or more processors communicatively coupled to the camera, the display, and the memory, the one or more processors configured to: capture, by the camera of the mobile device, the image of the live scene; and display, in the display of the mobile device, an augmented image based on the captured image, the augmented image comprising: traffic conditions, estimated time to a destination, route-related information, or any combination thereof; at least a portion of the image of the live scene; and a map configured to indicate one or more navigation instructions toward the destination, wherein the map is docked to one or more features of the live scene.
 8. The mobile device of claim 7, wherein the map is independently adjustable to the live scene, wherein independently adjustable comprises enlarged, reduced, panned, translated, oriented, or any combination thereof.
 9. The mobile device of claim 7, wherein the one or more features are user-defined, user-selected, or any combination thereof.
 10. The mobile device of claim 9, wherein the one or more processors are further configured to receive user input indicating the one or more features to use for docking the map.
 11. The mobile device of claim 9, wherein the map is overlaid in the live scene over an area associated with the one or more features.
 12. The mobile device of claim 7, the one or more processors are further configured to: obtain, via one or more transceivers, one or more augmented reality (AR) data associated with the location of the mobile device; and display, via the display, the one or more AR data in the augmented image based on one or more second features detected in the live scene.
 13. A mobile device comprising: means for capturing an image of a live scene; means for determining a location of the mobile device; and means for displaying an augmented image based on the captured image, wherein the augmented image comprises: traffic conditions, estimated time to a destination, route-related information, or any combination thereof; at least a portion of the image of the live scene; and a map configured to indicate one or more navigation instructions toward the destination, wherein the map is docked to one or more features of the live scene.
 14. The mobile device of claim 13, wherein the map is independently adjustable to the live scene, wherein independently adjustable comprises enlarged, reduced, panned, translated, oriented, or any combination thereof.
 15. The mobile device of claim 13, wherein the one or more features are user-defined, user-selected, or any combination thereof.
 16. The mobile device of claim 15, the mobile device further comprising means for receiving user input indicating the one or more features to use for docking the map.
 17. A non-transitory computer-readable medium for displaying an augmented image, the non-transitory computer-readable medium comprising processor-executable program code configured to cause one or more processors to: capture an image of a live scene; determine a location of a mobile device; and display the augmented image based on the captured image, wherein the augmented image comprises: traffic conditions, estimated time to a destination, route-related information, or any combination thereof; at least a portion of the image of the live scene; and a map configured to indicate one or more navigation instructions toward the destination, wherein the map is docked to one or more features of the live scene.
 18. The non-transitory computer-readable medium of claim 17, wherein the map is independently adjustable to the live scene, wherein independently adjustable comprises enlarged, reduced, panned, translated, oriented, or any combination thereof.
 19. The non-transitory computer-readable medium of claim 17, wherein the one or more features are user-defined, user-selected, or any combination thereof.
 20. The non-transitory computer-readable medium of claim 17, wherein the processor-executable program code is further configured to cause the one or more processors to: obtain one or more augmented reality (AR) data associated with the location of the mobile device; and display the one or more AR data in the augmented image based on one or more second features detected in the live scene. 